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It starts out innocently enough - users want to monitor Online data and so run their own copies 
of the detector control GUIs in their offices and at home. But over time, the number of processes 
making requests for values to display on GUIs, webpages and stripcharts can grow, and affect the 
performance of an Input/Output Controller (IOC) such that it is unable to respond to requests from 
requests critical to data-taking. At worst, an IOC can hang, its CPU having been allocated 100% 
to responding to network requests. 

For the BaBar Online Detector Control System, we were able to eliminate this problem and make 
great gains in security by moving all of the 10 Cs to a non-routed, virtual LAN and by enlisting a 
workstation with two network interface cards to act as the interface between the virtual LAN and 
the public BaBar network. On the interface machine, we run the Experimental Physics Industrial 
Control System (EPICS) Channel Access (CA) gateway software (originating from Advanced Photon 
Source). This software accepts as inputs, all the channels which are loaded into the EPICS databases 
on all the IOCs. It polls them to update its copy of the values. It answers requests from applications 
by sending them the currently cached value. 

We adopted the requirement that data-taking would be independent of the gateway, so that, in the 
event of a gateway failure, data-taking would be uninterrupted. In this way, we avoided introducing 
any new risk elements to data-taking. Security rules already in use by the IOC were propagated to 
the gateway's own security rules and the security of the IOCs themselves was improved by removing 
them from the public BaBar network. 
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I. INTRODUCTION 

This paper describes the motivation behind and im- 
plementation of a CA Gateway |l| in the BaBar Online 
Detector Controls System. 



II. MOTIVATION 

With the IOCs on the BaBar public network, there 
were no limits on the number of clients (aside from 
the limit of 384 ssh sessions) that could connect and 
request values. These clients could run on any of the 
63 workstations on the BaBar public network. Remote 
access to the BaBar public network was possible via 
ssh. 

As a result, the number of open file descriptors on 
an IOC could exceed the 150 limit (already increased 
significantly from the default 50) and the CPU usage 
of an IOC could go as high as 100 



III. IMPLEMENTATION 

The IOCs were moved to a private network and the 
four servers running EPICS client software that were 
essential for data-taking were given a secondary net- 
work interface so that they could access the private 
network. CA gateway software was installed on one 
of these servers, "bbr-srvOl". Statistics of the gate- 
way are displayed instantaneously in Figure^^nc! over 
time in Figure 

The gateway acts as a client to the IOCs, requesting 



Gateway bbr-srvO 1 

Active: 15692 
Alive: 17854 
Total VCs: 15692 
Total PVs: 18064 
FDs: O 
Client Event Rate : 2 88. 6 1 

Client Post Rate: 28 1 .42 

Exist Test Rate: 13.99 
Loop Rate: 91.71 
Server Event Rate: 1343.56 
Server Post Rate: 1343.56 

FIG. 1: CA gateway statistics. 



channels and receiving replies at the rate of approxi- 
mately 285 channels per sec (Figure [IJ. 

The gateway also acts as a server, providing a 
cached superset of all the IOCs' values to public 
EPICS clients. In Figure d we see that the number of 
channels active in the current cache is 15692 and the 
number of channels that will be held in the cache for 
up to two more hours is 17854. 

The server is receiving requests and responding to 
them at the rate of 1343.56 channels per second. The 
software was built without the flags to monitor the 
number of file descriptors in use, so this explains 
"FDs: 0". 

Plotted against time (Figure EJ), we see that the 
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FIG. 2: CA gateway statistics as a function of time. 



out which actions. For example, the access security 
group "dchexpert" may contain a list of users who 
are allowed to turn on the high voltage for the drift 
chamber. A security access file for the gateway was 
constructed from the sum of the individual IOC ac- 
cess security files. The user id running the gateway 
processes had to be added to the individual IOC's se- 
curity rules since requests to do secure transactions at 
the IOC level were being made by the gateway process 
and not the expert user, if initiated from the public 
network. 



number of alive and active channels is near constant 
and that there are small fluctuations in the event and 
post rates. 

The dual network interface card (NIC) servers re- 
quired a special EPICS configuration to avoid seeing 
ambiguous replies to their request for data (since they 
could access both the IOCs directly and the CA gate- 
way for values). 

The BaBar private network is not routed, so IOCs 
are hidden from the public internet address space. 

IOCs use security access files to define access secu- 
rity groups and rules to specify which users can carry 



IV. RESULTS 

The number of file descriptors decreased by 25% 
and the CPU usage decreased by 20-40%, on av- 
erage for the 17 IOCs. There were fewer IOC 
hangs/disconnects which helped to improve BaBar's 
data-taking efficiency. 

Internal IOC security is maintained by propagating 
the IOC security access definitions to the gateway se- 
curity definitions. External IOC security is improved 
since IOCs are no longer publicly visible. 



[1] |http: / / www-csr .bessy.de / con trol / SoftDist / Gateway | 



APPENDIX 



The poster displayed at CHEP 2003, in two halves 
(Figures |3| and HJ). 



Improving the Security and Performance of the BaBar Detector Controls System 
at the Stanford Linear Accelerator Center 



It starts out innocently enough - users want to monitor Online data and so run 
their own copies of the detector control GUIs in their offices and at home. But 
over time, the number of processes making requests for values to display on 
GUIs, webpages and stripcharts can grow, and affect the performance of an 
Input/Output Controller (IOC) such that it is unable to respond to requests 
from requests critical to data-taking. At worst, an IOC can hang, its CPU 
having been allocated 100% to responding to network requests. 

For the BaBar Online Detector Control System, we were able to eliminate this 
problem and make great gains in security by moving all of the lOCs to a 
non-routed, virtual LAN. By enlisting a workstation with two network 
interface cards to act as the interface between the virtual LAN and the public 
BaBar Network. On the interface machine, we run the Experimental Physics 
Industrial Control System (EPICS) Channel Access (CA) gateway software 
originating from Advanced Photon Source (APS). This software accepts as inputs, 
all the channels which are loaded into the EPICS databases on all the lOCs. It 
polls them to update its copy of the values. It answers requests from applications 
by sending them the currently cached value. 

We gave ourselves the requirement that data-taking would be independent of the 
gateway, so that, in the event of a gateway failure, data-taking would be 
uninterrupted, In this way, we avoided introducing any new risk elements to 
data-taking, Security rules already in use by the IOC were propagated to the 
gateway's own security rules and the security of the lOCs themselves was / 
improved by removing them from the public BaBar network. / 
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On every workstation running CA clients.. 



Outgoing UDP broadcast 



21 wkstns used by operators 
to control detector 





incoming^JDf^oaayast 
from each IOcC i ( 
(beacon) \\ 

TCP/IP connection to obtain PV 



caRepeater process on each wkstn: 

- listens for CA Server beacons on UDP port 

- fans out beacons to registered CA clients 
on this wkstn (not drawn) 

"A client application: 

■ broadcasts a request on UDP EPICS_CA_SERVER_PORT 

for Process Variable (PV) 

to list of CA Servers in EPICS_CA_ADDR_LIST 

receives UDP broadcasted answer to PV request 

initiates TCP/IP connection to CA server with the PV 

keeping connection open until beacons lapse beyond timeout 

or until ca_clear_channel is called for all PVs on that server 

Socket is reused if other PVs needed from same CA server 



, incoming UDP broadcast 
\ from IOC 
.)..... (PV) 



Sam pie Channel Access (CA) control client 



' Incoming request for PV 

on UDP EPICS CA SERVER^PORT _ , ,. . , , 

BaBar Public Network 



42 wkstns used by experts 
to monitor operations, test and 
\\ make modifications 



(up to maximum 384 remote ssh sessions) 



TCP/IP connection to obtain PV 



CA Server onXeaeh IOC 
broadcasts begcons every 1 5 s 
on UDP EPICS_OA_REPEATER_PORT 
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Sample Channel Access (CA) monitoring client 



CA Server on each IOC 
"\ broadcasts that it has PV 
\ \ on UDP EPICS_CA_SERVER_PORT 



Control Hardware and Instrumentation 
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Pathway to Main Control Center 



3 wkstns used by operators to control detector 

- no dependency on gateway 

- they talk directly to lOCs for PVs 

- configured not to see Gateway wkstn / 



TCP/IP connection to transfer PV 
frorr gateway cache 



17 Input/Output Controllers (lOCs) runningvxWorks 



^JGPTlP connection toffansfer PV 
to controls wfetn \ s v 



Detector Controls 
Private Network 





42 wkstns used by experts to monitor operations, 
xtest and make modifications 



CA client broadcast request for PV 



BaBar Public Network 



CA Server on each IOC\ 
broadcasts beacons every T5-s 

TCP/IP COTH 

on UDP EPICS CA REPEATEr 



I The Two Sides of the 
/Gateway Workstation/ 

acts as CA client i acts as CA server 
caching available PVs j serving PVs tc clients/A ^ ewQy broadcasts that it nQS pv 



tneclion toJtanSfer PV 
gateway cache 



Control Hardware and Instrumentation 

sun4u spare SUNW, Ultra -60 
gateway process uses 5-10% CPU 



Instantaneous gateway behaviour 



number of channels currently requested 
number of channels recently requested 
rates at which CA Clients are requesting (Hz) 

rates at which CA Servers are responding (Hz) 






1 8 wkstns used by operators to control detector 



TCP/IP connection to transfer PV 
from gateway cache 



CA client broadcast request for PV 



Benefits 




Security 


Performance 


CA Security is in place at two levels: 


CPU load on lOCs reduced by 20-40% 


Incoming write request from BaBar Public CA client app 


# open file descriptors on lOCs reduced by 25% 


is verified by gateway's security rules. If permitted, request 


fewer IOC hangs/disconnects 


is passed to IOC for internal verification 




Network security is improved 




lOCs not visible from BaBar Public Network 
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FIG. 4: CHEP03 poster contd. 



